On the Relationship Between Image and Motion Segmentation

In this paper we present a generative model for image sequences, which can be applied to motion segmentation and tracking, and to image sequence compression. The model consists of regions of relatively constant color that have a motion model explaining their motion in time. At each frame, the model can allow accretion and deletion of pixels. We also present an algorithm for maximizing the posterior probability of the image sequence model, based on the recently introduced Swendsen-Wang Cuts algorithm. We show how one can use multiple cues and model switching in a reversible manner to make better bottom-up proposals. The algorithm works on the 3d spatiotemporal pixel volume to reassign entire trajectories of constant color in very few steps, while maintaining detailed balance.